Skip to content

fix(plugin-profiles): prove the per-hook setup question is asked and carry review fixes - #16507

Merged
kyle-sexton merged 9 commits into
mainfrom
fix/plugin-profiles-followups
Nov 16, 2026
Merged

kyle-sexton merged 9 commits into
mainfrom
fix/plugin-profiles-followups

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

Refs: #16434

Summary

#16434 stayed open after #16483 because the per-hook setup case answered its own question in the prompt, so the headless session never had to ask it live. This PR rewrites that case so one headless claude -p session can show the question being asked. It also carries seven review suggestions left from the last pr-review-main verdicts on #16450 and #16483.

Fix

  • Per-hook setup case (plugins/plugin-profiles/evals/setup-asks-per-hook-plugin-and-saves-user-profile/): the prompt answers the work interview and the recording offer, but leaves notes-hooks' always-on or per-profile choice open. The new graders read:

    • per-hook-question-asked: the final reply names notes-hooks and asks always on or per profile.
    • no-profile-saved-before-answer: fails any profiles.mjs save … --yes.
    • The LLM grader fails a recorded or guessed answer.

    The three save graders and their samples are dropped. Saving at user level was proven by the three headless sessions on 03e149a (feat(plugin-profiles)!: switch plugin sets by profile and remove the core bundle #16483).

  • profiles.mjs check: a malformed user or repository profile is reported by its path, and the check still exits 0; a malformed shipped profile still exits 2.

  • scripts/check-plugin-profiles.sh: the gate runs with an empty temp home and config folder, so the caller's own profiles can no longer change its result.

  • check-skill.sh check 27: a ## Next gated on plugins that "are enabled" or "is not enabled" now warns, as "is enabled" already did.

  • Seam fallbacks: plugins/autonomy/reference/routines/drift-delta-sweep.md and plugins/verification/skills/confirm/SKILL.md say "not enabled" instead of "absent", and name the enable remedy. In the autonomy file the remedy is named by surface class, since its reference contracts ban vendor names.

  • scripts/cross-plugin-source-registry.txt: the comment now names where each copy sits.

  • Not changed:

    • skills/profile/SKILL.md ## Next: not a defect. .claude/rules/skill-bodies-state-current-rules.md says a Next section carries no availability gate.
    • plugins/testing/evals/run-e2e-route-install-suggestion/case.yaml: its tightened grader matches run-e2e/SKILL.md and its samples. Its LLM grader has not been run since that change.

Verification

  • Tests were written first and seen failing, then passing:
    • check-skill.test.sh: all assertions passed.
    • profiles.test.mjs with lib/check.test.mjs: 77/77.
    • check-plugin-profiles.test.sh: 4/4.
  • validate-cases.py on plugins/plugin-profiles/evals: only the existing allowed_tools warnings. An up-front-answer reply and a save --yes call, offered as must-pass samples in a scratch copy, were both rejected.
  • scripts/check-plugin-profiles.sh: passes against the catalog. Pre-push check-affected: passes.

Remaining before Closes #16434: one headless claude -p session for this case, run in a throwaway config after the owner signs in to it. It is graded by the case's graders, and the config is logged out afterwards.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_015XWwmfnVygLgeDpAXkfoJp

kyle-sexton and others added 7 commits November 15, 2026 23:52
… is asked

The case prompt stated the always-on or per-profile answer up front, so a
pass showed the answer recorded, not the question asked. The prompt now
answers the work interview and the recording offer only, and the graders
check that the final reply asks about notes-hooks and that no profile is
saved with --yes before the answer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfnVygLgeDpAXkfoJp
…in check

check stopped with exit 2 on a malformed user or repository profile,
though those levels are reported, never failed. It now reports the file
and its error and checks the other levels; a malformed shipped file still
stops the check with the file named.

scripts/check-plugin-profiles.sh now runs the check with an empty home
and config folder and the checked tree as the project, so the caller's
installed plugins and own profiles never reach the CI result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfnVygLgeDpAXkfoJp
…ction

The operative-phrasing check matched "is|if|when enabled" only, so a
"## Next" gated on plugins that "are enabled" or covering one that "is
not enabled" passed as a plain mention.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfnVygLgeDpAXkfoJp
… remedy

The drift delta sweep and /verification:confirm Stage 1 gated on a plugin
being enabled but took the fallback when it was "absent". Per the
seam-phrasing contract the fallback covers a plugin that is not enabled,
and the report names claude plugin enable, preceded by claude plugin
install when the plugin is not installed.

The plugin-options cluster comment in the cross-plugin source registry
said every other copy sits at a path of its own; the harness-ops and
plugin-profiles copies sit at the canonical path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfnVygLgeDpAXkfoJp
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 05:15
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T05:18:15.482316Z 2c7df0c Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-refine/fix-docs
Model rung 0
Leg handoff
Run job success
Activity step success
Commits pushed none (read lane)
Run 43809067731

Failed GitHub API calls

Method Path Status
GET /repos/{repo}/git/ref/tags/* 404

Lane-Activity: pr-refine/fix-docs
@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-refine/apply-patch
Model rung 0
Leg base
Run job success
Activity step success
Commits pushed effa5ae
Run 43809067731

Failed GitHub API calls

Method Path Status
GET /repos/{repo}/git/ref/tags/* 404

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed ba5b5472701758f85e64e190ee051d228e8390aa: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #16507, head ba5b547 (incremental since effa5ae)

No security issues found. This was an incremental review, limited to blocking findings only (the lane already reviewed an earlier head). I read every hunk of both files in scope:

  • plugins/plugin-profiles/evals/setup-asks-per-hook-plugin-and-saves-user-profile/prompt.md: AskUserQuestion is removed from allowed_tools, so the line now reads allowed_tools: [Skill, Bash, Read, Write]. This takes a tool away from the eval session and adds none, so it widens nothing. runs: 1, max_turns: 30 and timeout_seconds: 600 are unchanged.
  • .../graders/per-hook-question.md: the LLM grader now expects the question to be asked "in its reply" rather than "in its reply or an AskUserQuestion call". This wording matches the tool removal. The fail conditions are unchanged, including the one that fails any profile saved with --yes before the user answers.

Neither file matches a trusted-path glob. No plugin manifest changed and no standing instruction was weakened. Both files are eval fixtures and reach no credentialed lane.

Spend: no billing impact. The eval's run budget is unchanged.

critical-findings: 0

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on effa5ae857bbbd188d639ee8775ee93102379e76; the summary is in this lane's pull request comment.

claude-lane: claude-review

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed ba5b5472701758f85e64e190ee051d228e8390aa: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #16507 (incremental, effa5ae..ba5b547, blocking only)

I read every hunk of both files in the incremental scope:

  • plugins/plugin-profiles/evals/setup-asks-per-hook-plugin-and-saves-user-profile/graders/per-hook-question.md
  • plugins/plugin-profiles/evals/setup-asks-per-hook-plugin-and-saves-user-profile/prompt.md

This lane has reviewed an earlier head, so this pass reports blocking findings only.

What this push changes: it applies the second fix offered in this lane's earlier finding (discussion_r4582274119).

  • AskUserQuestion is gone from the case's allowed_tools.
  • The LLM grader no longer accepts "or an AskUserQuestion call".

The per-hook-question-asked regex grader reads last_message. Both graders now expect the always-on or per-profile question in the reply text, so they no longer disagree on how the question was asked. I searched the case directory and found no other mention of AskUserQuestion.

Settled: that thread is resolved. The owner's reply states the fix, and the job reports no line changes since resolution. I am not raising it again.

Not verified: I ran nothing. The validate-cases.py result in the owner's reply is the author's claim. The headless claude -p run is still needed before this PR closes #16434.

Closing issue: the PR only refers to #16434 (Refs:) and closes no issue, so there were no acceptance criteria to check.

Manifests: none changed.

Spend: removing a tool from an eval case only affects prepaid model usage when the case runs. It is not a billing change.

No new findings. Verifier dropped 0 of 0 candidates.

…in the reply

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015XWwmfnVygLgeDpAXkfoJp
@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 9b747fa Nov 16, 2026
54 checks passed
@kyle-sexton
kyle-sexton deleted the fix/plugin-profiles-followups branch November 16, 2026 05:52
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Post-merge mutation testing

Killed 5, survived 2, timeout 0, no coverage 0, invalid 0, not run 0.
Covered-code score 71%; mutation score 71%.

File Killed Survived Timeout No coverage Invalid Not run
plugins/plugin-profiles/scripts/profiles.mjs 3 1 0 0 0 0
plugins/skill-quality/scripts/check-skill.sh 1 0 0 0 0 0
scripts/check-plugin-profiles.sh 1 1 0 0 0 0

Surviving mutants

Location Operator Disposition Cause
plugins/plugin-profiles/scripts/profiles.mjs:597 LCR equivalent none
scripts/check-plugin-profiles.sh:18 SBR productive no-assertion

The run's summary has the full report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant